Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The OpenVM v2 CPU prover relies on SIMD auto-vectorization which is
killed by codegen-units=256. Reduce to 16 and add thin LTO.
Also enable AOT feature for faster execution.

OpenVM v2 doesn't need timestamp-based segmentation - the timestamp
no longer overflows without it. Removing the delta avoids the 4824
segment explosion that was causing keccak_prove_large to take forever.

The timestamp reset fix in OpenVM makes the delta work correctly
again. Without the delta, proofs may be unsound due to timestamp
overflow within a segment.

The OpenVM default max_trace_height (2^22, ~4 GiB per chip) made
concurrent GPU tests contend for device memory on the shared runner.
Tests here never need that height, so default them to 2^20 and also
apply the override to the mock path via `do_with_trace`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… memory

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…m-v2-integration-beta.2

# Conflicts:
#	.github/workflows/build-cache.yml
#	.github/workflows/nightly-tests.yml
#	.github/workflows/post-merge-tests.yml
#	.github/workflows/pr-tests-with-secrets.yml
#	.github/workflows/pr-tests.yml

- sha256 -> sha2 in openvm.toml configs
- sha256() -> Sha256::new/update/finalize
- Add openvm-bigint-guest dep for u256 intrinsics
- Fix branch refs to v2-powdr-beta.2

- Remove custom_pvs field (leftover from failed conflict resolution)
- Use stark-backend tag instead of branch for stable reference

Replace MappingRowEvaluator's BTreeMap<u64, usize> with dense Vec<usize>
indexed by polynomial ID. Eliminates ~2 billion O(log n) BTreeMap lookups
in the hot bus interaction replay loop.

Result: powdr_bus_replay 36.7s -> 27.0s (1.36x speedup)
        trace_gen total  58.5s -> 41.4s (1.41x speedup)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Pre-compile AlgebraicExpression ASTs to CompiledExpr enum at build time:
- Constant(F): zero-cost eval
- DirectLoad(usize): single array access
- LinearCombination{constant, terms}: tight multiply-accumulate loop
- General: fallback AST walk with pre-resolved column indices

Eliminates recursive tree walk, closure dispatch, and enum matching
from the per-row bus interaction replay loop.

Result: powdr_bus_replay 27.0s -> 3.3s (8.3x speedup over O1)
        trace_gen total  41.4s -> 17.4s (2.38x speedup over O1)
        vs original baseline: 58.5s -> 17.4s (3.36x total speedup)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Switch the per-row loop from sequential .chunks_mut/.for_each to
parallel .par_chunks_mut/.for_each. Periphery chips already use
AtomicU32 counters, so no thread-local accumulators needed.

Extract periphery references to avoid capturing self (ISA::Config
is not Sync).

Result: fill_rows_and_replay 5.1s -> 1.9s (2.6x from parallelization)
        trace_gen total  17.4s -> 13.1s (1.33x over O1+O2)
        vs original baseline: 58.5s -> 13.1s (4.45x cumulative)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Remove "O2 optimization" label from comment
- Add detailed doc comment on CompiledExpr explaining why specialized
  variants exist (avoid recursive dispatch, closure overhead, and
  trait-object cost of AlgebraicExpression::to_expression)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove info_span! wrappers, metrics gauge emissions, and timing
code that were used during optimization development. The actual
optimizations (Vec lookup, CompiledExpr, rayon parallelization)
remain unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…on comment

Explains why par_chunks_mut is now safe: periphery chips use AtomicU32
counters, so concurrent apply() calls are thread-safe. The original
code used serial chunks_mut because apply() was not thread-safe.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Shadows the BTreeMap with the dense Vec of the same name, reducing
diff noise from the original code.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add ArenaType enum to select between MatrixRecordArena (CPU) and
DenseRecordArena (GPU) byte layouts. The key difference is core
record byte offset:
- Matrix: adapter_cols * sizeof(F) (e.g., 76 for BaseAlu)
- Dense: aligned_adapter_record_size (e.g., 40 for BaseAlu)

All 4 AIR mapping functions now accept ArenaType parameter.
Existing CPU path uses ArenaType::Matrix (unchanged behavior).
GPU path will use ArenaType::Dense.

CPU validation still passes: 48,480 columns match.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- apc_jit_tracegen.cu: data-driven CUDA kernel that reads DenseRecordArena
  bytes directly and writes APC columns via descriptor arrays.
  Supports all 27 computation types (DirectU32/U8/U16, TimestampDecomp,
  AluResult, ShiftResult, PointerLimb, BranchEqualCmpResult, LoadStore
  flags/write_data, etc.) with conditional variants via high-bit flag.
- cuda_abi.rs: JitColumnDesc/JitInstructionDesc FFI structs + safe wrapper
- cuda/mod.rs: try_generate_witness_jit() replaces Stage 1+2 with single
  JIT kernel launch. Keeps existing derived_expr + bus kernels for Stage 3.
- POWDR_JIT_TRACEGEN=1 activates GPU JIT path with graceful fallback.
- Verified: real GPU prove passes on keccak.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add if/else fast-path before the switch for DirectU32/U8/U16 columns
(~88% of all columns) to avoid the full switch dispatch.

Result: NO improvement. 13135ms vs 13045ms original JIT (within noise).
GPU branch prediction doesn't benefit from this pattern — all threads
in a warp process the same descriptor sequence with mixed types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This reverts commit 11d5915.

Previous GPU benchmarks were actually running CPU path because
cli-openvm-riscv didn't have cuda feature. Add cuda to default features.

Real GPU comparison (10k keccak):
  Baseline: 835 ms total trace gen (9.6s total prove)
  JIT:      2909 ms total trace gen (10.5s total prove)
  JIT is 3.5x slower on GPU trace gen.

The switch-based eval_jit_column dispatch is significantly slower
than the pre-compiled per-AIR-type CUDA kernels.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…verhead)

Real GPU benchmarks (10k keccak, cuda feature properly enabled):
  Baseline total trace gen: 835 ms (per-seg: 281, 209, 207, 138)
  JIT total trace gen:      2909 ms (per-seg: 729, 726, 731, 723)

Root cause: the descriptor-driven eval_jit_column switch (24 cases)
is fundamentally slower than pre-compiled per-AIR CUDA kernels.
The baseline's per-AIR kernels are monomorphic and extremely fast
(LoadStore: 8ms, BaseAlu: 3ms per segment).

The data-driven approach cannot match pre-compiled native code.
Next step: NVRTC runtime compilation to generate native per-APC
CUDA kernels with no switch dispatch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Persists compiled PTX bytes to ~/.cache/powdr/nvrtc_kernels/<hash>.ptx
keyed by source_hash (which already includes EMITTER_VERSION). On compile,
checks disk first; on miss, runs NVRTC and atomically writes the PTX. Loads
go via cuModuleLoadData directly, skipping NVRTC entirely.

Cold start (input=10000 keccak APC=1, 2022 cols, ~387KB source): ~96-115s.
Warm start: ~5.7ms PTX load. Persisted across processes.

Env: POWDR_NVRTC_CACHE_DIR overrides; POWDR_NVRTC_NO_DISK_CACHE disables.
POWDR_NVRTC_DUMP_DIR (preexisting) writes the .cu source for debugging.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The descriptor-JIT and NVRTC paths were spending 590ms (out of 736ms total
trace_gen on 10k keccak APC=1) in a pure-overhead host memcpy:
extend_from_slice'ing 1.65 GB of arena bytes from per-AIR DenseRecordArena
buffers into a single Vec<u8> before H2D. That host concat was structural
waste — cudaMemcpyAsync does not require contiguous host source.

Replaced with concat_arenas_direct_h2d: allocate one DeviceBuffer<u8> of
total size, then issue one cudaMemcpyAsync per arena directly to its
offset. Same total bytes uploaded, no host-side intermediate. Verifier
PASSes on keccak APC=1 (2022 cols x 2400 rows match descriptor backend).

Sub-stage profiling under POWDR_TRACE_PROFILE=1 now wraps every major
step in try_generate_witness, _jit, _nvrtc with stream-synced timing
(arena_concat_host/arena_direct_h2d/arena_h2d, build_descriptors,
descriptors_h2d, allocate_zero_output, surviving_kernel,
derived_compile_h2d, derived_kernel, bus_compile_h2d, bus_kernel,
fill_dummy_traces, dummy_chip_inventory, build_subs, subs_h2d,
emit_source, compile_or_cache_load).

10k keccak APC=1, single segment, mock prove (post-fix):
  baseline trace_gen     263 ms (fill_dummy_traces 218 ms incl. internal
                                  H2D + per-AIR fill_trace_row)
  descriptor trace_gen   184 ms (arena_direct_h2d 140 ms; 1.43x faster
                                  than baseline)
  nvrtc warm trace_gen   197 ms (arena_direct_h2d 144 ms; 1.34x faster)

Both JIT paths are now faster than the substitution baseline at the
trace_gen wall-time level. The arena H2D is the dominant remaining cost
(~73% of NVRTC trace_gen). Per-AIR fill_trace_row also does its own H2D
(see base_alu/cuda.rs records.to_device()), so the H2D is structural in
both paths; baseline pays it inside each chip's generate_proving_ctx.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…armup probe

Adds three new POWDR_TRACE_PROFILE-gated measurements to the baseline path:
- baseline.measure_h2d_only: pure 1-alloc-N-memcpy upload of all APC arenas
  before any chip.generate_proving_ctx work, for direct comparison to JIT's
  arena_direct_h2d. Same 1.65 GB, same code path; baseline saw 41 ms while
  JIT sees 105-145 ms (cause unidentified after testing VPMM warmup, 1 MB
  H2D warmup, and 256 MB H2D warmup — none affected JIT timing).
- per-AIR fill_dummy_trace[APC/<air>] and APC/non-APC totals: time each
  chip.generate_proving_ctx individually (covers H2D + fill_trace_row +
  alloc), tagged by whether the AIR is referenced by the APC's instruction
  list. Confirms baseline only does H2D for APC AIRs (1.65 GB total); non-
  APC AIRs (Mult, HintStore, etc.) generate traces outside this code path.
- maybe_warmup_vpmm: env-gated POWDR_VPMM_WARMUP that pre-allocates a
  buffer at the target size and does a probe H2D to pay any first-use
  cost (VPMM cuMemMap, pinned staging init). Empirically, this had no
  effect on arena_direct_h2d timing.

10k keccak APC=1, single segment, mock prove:
  baseline.dummy_chip_inventory                  0.275 ms
  baseline.measure_h2d_only           1.65 GB   41.0 ms
  baseline.fill_dummy_trace[APC/JalLui]                  0 B    0.001 ms
  baseline.fill_dummy_trace[APC/BranchEqualCore]    2.6 MB    0.6 ms
  baseline.fill_dummy_trace[APC/LoadStoreCore]      393 MB   29.1 ms
  baseline.fill_dummy_trace[APC/ShiftCore]          368 MB   29.7 ms
  baseline.fill_dummy_trace[APC/BaseAluCore]        886 MB  124.7 ms
  baseline.fill_dummy_trace_APC_total            1.65 GB  185.2 ms
  baseline.fill_dummy_traces (wrapper)                    219 ms
  baseline.bus_kernel                                      20.9 ms
  trace_gen total (segment 0)                             263 ms

  descriptor.arena_direct_h2d         1.65 GB  105-145 ms (variance)
  descriptor.bus_kernel                          21 ms
  trace_gen total                                184-197 ms

JIT's arena_direct_h2d remains 100ms slower than baseline's
measure_h2d_only for the same 1.65 GB transfer through the same code path.
Cause unknown; deferred for further investigation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…emory

When per-call sync timing showed only 42 ms of cudaMemcpyAsync work for
1.65 GB but the arena_direct_h2d wrapper measured 105+ ms, the missing
60 ms was inside the function but outside per-call brackets. Adding
explicit drop-time measurement for the consumed `arenas: Vec<(String,
DenseRecordArena)>` reveals the answer:

  concat.fn_pre_drop           41.2 ms  (alloc + 4 cudaMemcpyAsync at ~39 GB/s each)
  concat.arenas_drop           65.5 ms  (host-side free of 1.65 GB pageable memory)
  concat.fn_total_to_return   106.7 ms
  time_stage wrapper          106.7 ms  (matches)

The 65 ms is glibc free()/munmap of 4 large Vec<u8> chunks (886 MB +
393 MB + 368 MB + 2.6 MB) inside DenseRecordArena structs. It's real
host work, not a measurement artifact.

This explains the apparent baseline-vs-JIT H2D discrepancy:
  - baseline.measure_h2d_only    = 42 ms  (pure H2D; arenas are borrowed,
                                            not dropped during measurement)
  - descriptor.arena_direct_h2d  = 107 ms (H2D + arena Vec drop)

In baseline, arena drops are distributed across per-chip
generate_proving_ctx calls (~60 ms total spread across BaseAlu/LoadStore/
Shift). In JIT, all arenas drop together at end of concat. Both paths
pay ~107 ms total for arena handling (H2D + free) — apples-to-apples
confirmed once both costs are accounted for.

Per-call profile gated on POWDR_PER_CALL_PROFILE=1 (in addition to the
existing POWDR_TRACE_PROFILE).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Unblocks keccak APC=10, which previously triggered "no mapping for AIR"
fallback warnings (4× LessThan, 8× BranchLessThan across 4 segments) and
forced those chips through the per-AIR fill_trace_row substitution path.

Descriptor backend only — NVRTC emitter returns None for the new arms so
APCs that include them fall back to descriptor per-APC.

11 new ColumnComputation variants, all with uniform byte-offset signatures
so a future NVRTC fusion pass can group columns sharing (opcode, b, c)
into one run_less_than / run_branch_lt call:

  LessThan{CmpResult, DiffVal, DiffMarker, BMsbF, CMsbF}
  BranchLt{CmpResult, CmpLt, DiffVal, DiffMarker, AMsbF, BMsbF}

CUDA kernel additions: 11 new JitCompType cases (24-34) + two device
helpers run_less_than() / run_branch_lt() mirroring the chips' fill_trace_row
logic. Field-element computation matches the chip's signed-aware encoding
(b_msb_f = -Fp(256 - b[3]) when sign bit set in slt/blt mode).

Mappings:
  LessThan: Rv32BaseAluAdapter (10 cols shared with BaseAlu/Shift) +
            18 core cols → width 37, stride 52.
  BranchLt: Rv32BranchAdapter (10 cols shared with BranchEqual) +
            22 core cols → width 32, stride 40.

Smoke: prove --mock guest-keccak --autoprecompiles 10 --input 100 logs
"GPU JIT (descriptor) trace gen used" 10/10 times, 0 fallback warnings,
0 panics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

APC=30 surfaced two more missing AIRs:
  Rv32AuipcCoreAir       (auipc) — Rv32RdWriteAdapter, width 20
  Rv32JalrCoreAir        (jalr)  — Rv32JalrAdapter,    width 28

5 new ColumnComputation arms, all designed with NVRTC-fusion-friendly
byte-offset signatures so a future emitter pass can group columns sharing
the same record-byte inputs:

  AuipcRdLimb { pc_off, imm_off, limb_index }
       — (pc + (imm << 8)).to_le_bytes()[limb_index]
  JalrToPcLsb / JalrToPcLimb / JalrRdLimb
       — to_pc / rd_data field-element computations
  ConditionalNotMaxU32 { ptr_byte_offset, then_comp }
       — generic gate: evaluates then_comp iff *(u32*)(record + ptr) != u32::MAX
       — encoded as a new comp_type flag bit (0x40, JIT_COND_NOT_MAX_U32) to
         match the existing 0x80 byte-condition flag

CUDA additions: 4 new JIT_AUIPC/JALR_* switch cases + the NOT_MAX_U32
conditional flag. ConditionalNotMaxU32 reuses the same p[5] slot as
JIT_COND_FLAG; only one flag bit is set per descriptor.

Bug fixed during integration: I had naively reused LoadStoreWriteAuxDecomp
for Jalr's rd_aux ts_decomp, but that arm hardcodes delta=2 (LoadStore
does 1 read + 1 write at from_ts+1, then mem_helper.fill uses curr_ts =
from_ts+2). Jalr does 1 read + 1 write at from_ts+1, so delta=1. Replaced
with `ConditionalNotMaxU32 { then: TimestampDecomp { delta: 1, .. } }`.

Stride correction: Rv32JalrCoreRecord with #[repr(C)] is 16 bytes
(u16 imm + 2 pad + 2 u32 + bool + 3 pad, aligned to u32), not 12 as I
first assumed. Total Jalr stride is 28 + 16 = 44.

Smoke (real prove, no --mock):
  prove guest-keccak --autoprecompiles 30 --input 100
  baseline public_values_commit: [1877287944, 243417119, 1358120863, ...]
  descrip  public_values_commit: [1877287944, 243417119, 1358120863, ...]
  bit-identical → traces match fill_trace_row exactly.

mock APC=30 input=100 logs 30/30 "GPU JIT (descriptor) trace gen used",
0 fallback warnings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Phase 0 spike measurement (POWDR_BUS_SPIKE=1):
- Add apc_apply_bus_kernel_spike: same grid as real kernel but no
  bytecode VM, just 1 trace cell load + 1 atomicAdd per (interaction, row).
- APC=30 keccak: real bus_kernel 282ms (90 calls × 3.13ms), spike 7.5ms
  (30 calls × 0.25ms). Per-call ceiling 12.5x speedup → bus kernel is
  dispatch-bound, not atomic-bound. NVRTC will help.
- bus_kernel is 94% of trace gen time (surviving 18ms, derived 0.8ms).

Phase 1 refactor: extract run_bus_pass helper from three identical
inline call sites in try_generate_witness, _jit, _nvrtc. Centralize
POWDR_BUS_SPIKE branching. Prerequisite for Phase 2 NVRTC bus emitter
swap-in (one helper → one swap, not three).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…cal slice

New module nvrtc_bus_emit:
- BusEmitterInput + BusInteractionDesc model bus interactions per APC.
- BusKind classifies (VarRange / Tuple2 / Bitwise / Unsupported) at host emit
  time; only VarRange gets straight-line code in this phase.
- emit_bus_kernel_source produces a kernel with one switch arm per
  interaction. Each VarRange arm:
    * resolves mult/value/max_bits expressions to canonical u32 (Monty
      reduction) via emit_canonical_expr / emit_monty_expr — recursive AST
      walk yielding fresh-temp SSA-style C++.
    * unrolls mult when it is `Number(c)` (skip when c==0, single call when
      c==1, #pragma-unrolled loop when c>1); guarded runtime loop otherwise.
    * calls lookup::Histogram::add_count(idx) where idx = (1 << bits) + value - 1.
- Tuple2/Bitwise/Unsupported emit a `/* deferred */` comment so the structure
  stays stable for Phase 3+5 to fill in.
- Source hash mixes a literal "BUS" tag so a bus-kernel hash cannot collide
  with a trace-gen kernel hash even on identical structural inputs.

Self-contained PRELUDE inlines lookup::Histogram + monty_reduce + add/sub/
mul/neg_monty (NVRTC compile path doesn't add primitives include dirs, so
including primitives/histogram.cuh would fail).

Launcher: powdr_nvrtc_launch_bus_v1 in nvrtc_runtime.cu uses cuLaunchKernel
with the 12-arg bus_v1 signature; FFI decl in cuda_abi.rs.

Vertical-slice tests in nvrtc_bus_emit::tests:
- emits_var_range_only_kernel: source-shape sanity check.
- emits_skips_tuple_and_bitwise: verifies the deferred placeholder.
- host_to_monty_matches_known_values: monty(0)=0, monty(1)=R mod P.
- launch_var_range_only_kernel_updates_histogram: end-to-end emit→NVRTC
  compile→launch→D2H. Synthetic 8-row trace with value=row, max_bits=4 →
  asserts each of bins 15..22 incremented exactly once.
- launch_var_range_mult_three_unrolls: same but mult=3 → asserts each bin
  has 3 counts (proves the unroll path executes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Tuple-range emitter:
- BusKind::Tuple2 arm in emit_interaction now emits straight-line code
  matching openvm's RangeTupleChecker<2>::add_count: idx = v0 * sz1 + v1,
  histogram size sz0 * sz1.
- BusKind::Bitwise/Unsupported still emit /* deferred */ (Phase 5).
- EMITTER_VERSION bumped to 2 to invalidate any disk cache from Phase 2.
- Vertical-slice test launch_tuple_range_kernel_updates_histogram: 6
  rows of (v0%4, v1%8) → asserts each (v0,v1) pair counted exactly once.

Rayon parallel compile (NvrtcKernelCache::get_or_compile_many):
- Dedupes within batch by source_hash before compile, so duplicate
  kernels share an Arc.
- par_iter compile of unique misses; remaining work serial under the
  single Mutex.
- ensure_primary_context now uses thread_local init: cuCtxSetCurrent
  is per-thread, so each rayon worker must call it the first time. Was
  the root cause of Load { code: -2201 } INVALID_CONTEXT we saw on the
  initial parallel test run.
- Test cache_parallel_compile_dedupes_and_populates: 4 distinct kernels
  + 1 duplicate of the second; asserts the duplicate dedupes to the
  same Arc and that the cache is populated for follow-up get_or_compile
  calls.

Also: rayon dep added to openvm/Cargo.toml.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- BusKind::Bitwise arm now emits straight-line code matching openvm's
  BitwiseOperationLookup. BITWISE_NUM_BITS = 8 is fixed in the upstream
  kernel; total bins = 2 * 65536 (range half + xor half).
- Selector folding: when args[3] is `Number(c)`, host-folds to a single
  add_count call (the common case — each interaction is either always
  range or always xor). Otherwise emits the if/else dispatch inside the
  mult loop.
- args[2] (`x_xor_y`) is intentionally not consumed; the constraint
  system uses it but the histogram update doesn't.
- EMITTER_VERSION bumped to 3.

Vertical-slice test launch_bitwise_kernel_updates_both_halves: two
interactions over the same (x,y) trace cells, one selector=0 (range)
one selector=1 (xor). Asserts both halves of the 131072-bin histogram
are correctly populated and exactly 2*N bins are nonzero.

emits_bitwise_arms_inline replaces emits_skips_bitwise. New
emits_skips_unsupported_only covers the remaining deferred path.

All three bus types (var_range, tuple_range, bitwise) now have
straight-line emitters — Phase 4 verifier wiring + prove-path swap can
proceed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

run_bus_pass now dispatches on POWDR_JIT_BACKEND_BUS env:
- POWDR_BUS_SPIKE → spike kernel (Phase 0)
- POWDR_JIT_BACKEND_BUS=nvrtc → launch_nvrtc_bus
- default → existing bytecode-VM kernel (untouched)

launch_nvrtc_bus calls build_emitter_input → emit_bus_kernel_source →
NvrtcKernelCache::global().get_or_compile → powdr_nvrtc_launch_bus_v1
with the periphery histogram buffers. Skip the bus_compile_h2d step
in NVRTC mode since the kernel source replaces the bytecode/spans.

KNOWN ISSUE: this NVRTC backend does not yet work end-to-end on real
APCs. A keccak APC carries ~1734 bus interactions; the current emitter
produces a 30K-line kernel with a 1734-arm switch, and NVRTC compile
hangs (>10 min) on it. The wiring is in place gated off by default —
must NOT enable POWDR_JIT_BACKEND_BUS=nvrtc on real proves until the
emitter is rearchitected. See nvrtc_bus_phase0.md for the next step
(group-by-kind tight loops with op tables, instead of per-interaction
switch arms).

Phase 4 verifier (POWDR_JIT_BACKEND_BUS_VERIFY=1) is intentionally
left off: snapshot/restore D2H/H2D plumbing complicated enough to be
its own change, and the NVRTC backend has to compile in reasonable
time first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Restructure following independent reviewer's critique. Old emitter
produced one giant per-interaction switch (30K lines for keccak APC,
NVRTC compile hung >10min). New emitter:

- 4 fixed kernel sources (var_range, tuple2, bitwise_range, bitwise_xor)
  shared across all APCs and all runs. PTX disk cache has 4 files total
  instead of one per APC.
- Per-APC op tables are uploaded to __constant__ memory at launch time
  via cuModuleGetGlobal + cuMemcpyHtoD (new helper
  powdr_nvrtc_set_constant_symbol).
- Each Op struct includes mult_const + guard_col so we can express
  shapes like `mult = is_valid * 1` (collapses to mult_const=1,
  guard_col=is_valid). decode_mult handles Number, Reference, and
  Number*Reference forms via simplify_mult preprocessing.
- New add_count_n histogram primitive multiplies the warp-deduped
  popc-count by mult in a single atomicAdd. Avoids the
  for-k=0..mult-loop, which was unsafe under per-lane mult divergence
  on Volta+.
- Hybrid fallback: any interaction whose mult/args don't fit the
  simple form lands in PartitionedBus.unhandled and runs through the
  existing bytecode-VM kernel filtered to that subset. Both paths
  share the periphery histogram buffers.

New launcher powdr_nvrtc_launch_bus_v2 uses a uniform 6/7-arg layout;
the bitwise kernels declare an unused extra0 to keep the launcher
agnostic to kind.

Validation:
- All 13 cuda unit tests pass (including the 4-kernel-source compile via
  NvrtcKernelCache).
- Mock APC=10 keccak input=100 ✓
- Real APC=10 keccak input=100 produces public_values_commit
  bit-identical to the bytecode-VM baseline.
- Real APC=30 keccak input=10000 produces public_values_commit
  bit-identical to the bytecode-VM baseline (2c0a5e91 etc.).

Performance (RTX 5090, APC=30 input=10000):
- VM bus_kernel total: 281.5 ms (90 calls, avg 3.13 ms)
- NVRTC bus_kernel total: 308.5 ms (90 calls, avg 3.43 ms) — currently
  slower than VM. App proof end-to-end is within noise (8.24s vs 8.28s).

Speedup gap is a follow-up. Likely causes: (1) the 34% unhandled tail
still runs through bytecode VM and dominates; (2) 4 cuCtxSynchronize
syncs per APC; (3) keccak bus interactions with non-simple args
(`Number(c) - Reference(col)` etc) need richer arg-shape support to
shrink the unhandled fraction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Sub-stage timers (added to launch_nvrtc_bus):
- partition / kernel_prefetch / per-kind upload+launch / fallback split
  into compile_h2d + vm. Each wrapped in time_stage so POWDR_TRACE_PROFILE
  reports them.
- First-APC partition diagnostic now also dumps the first 5 unhandled
  interactions' shapes for debugging.

Result: revealed that fallback_vm is 82% of bus pass time (255 of 311ms),
because the 23-34% "unhandled" interactions are the EXPENSIVE ones
(timestamp deltas with multi-column affine expressions). The 4 NVRTC
kernels combined are 45ms / 14% — fast.

Affine arg support (decode_arg + Op-struct expansion):
- Each "data arg" (var_range value, tuple v0/v1, bitwise x/y) now
  encodes as `coef0 * d_output[col] + coef1` in the field. Coefficients
  baked at host time, stored in Montgomery form in the __constant__ op
  table. Kernel does mul_monty + add_monty + monty_reduce.
- Captures shapes: Reference, c-col, col-c, c+col, col+c, c*col,
  c*col+d, d+c*col, plus UnaryOperation::Minus (recursive).
- Op struct sizes: VarRangeOp 24B, Tuple2Op 32B, BitwiseOp 32B.
  MAX_OPS_PER_KIND lowered 4096 → 2048 to fit in 64KB constant memory.
- EMITTER_VERSION bumped to 5.

Measured impact (APC=30 keccak input=10000, RTX 5090):
- v4 (no affine):  unhandled=31, bus_kernel 311 ms
- v5 (with affine + Minus): unhandled=23, bus_kernel 315 ms
- The 8 newly-captured interactions are lightweight; the 23 remaining
  account for ~91% of fallback_vm cost (timestamp-delta evals).
- public_values_commit matches bytecode-VM baseline.

Next lever to capture remaining headroom: multi-column affine
(`c1*col_a + c2*col_b - c3*col_c + d`) — the timestamp-delta shape.
Requires expanding op struct further (likely store ops in global memory
since 4-term × 12B = 48B per op exceeds constant-memory budget at
1024+ ops).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…stogram

Replaces the take(5) "first 5 unhandled" dump with a full inventory:
- bytecode_size per interaction (proxy for VM per-row cost)
- max distinct columns referenced per arg (tells us if multi-col-affine
  decoder would cover it)
- structural shape_summary string (Number/Reference/op trees with leaves
  abbreviated, so identical SHAPES across different cols/constants
  collapse to one fingerprint)
- aggregations: by max_cols bucket, by shape histogram

Used to validate whether multi-column affine support is the right next
optimization step before committing to the 1-2 day implementation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Each "data arg" (var_range value, tuple v0/v1, bitwise x/y) now decodes
into N-term affine form `coef_const + sum_i coef_i * d_output[col_i]` for
up to MAX_TERMS_PER_ARG = 5 distinct columns. Captures keccak's
timestamp-delta shapes:
  - 22/23 unhandled fit in 3 terms (`c1*col_a + c2*col_b - c3*col_c + d`)
  - The 23rd is genuinely non-affine (Reference*Reference) → stays on
    bytecode VM.

Implementation:
- AffineArg struct: (n_terms, cols[5], coefs_monty[5], coef_const_monty)
  = 12 u32 / 48 bytes per arg. Embedded in VarRangeOp / Tuple2Op /
  BitwiseOp.
- decode_affine_arg with collect_affine recursive walker. Handles
  arbitrary nesting of Add/Sub/Number*Reference/UnaryMinus and merges
  same-column terms automatically.
- Op tables move from __constant__ to global memory: Op sizes
  (60-104 bytes) × 1000+ ops would exceed CUDA's 64KB-per-module
  constant memory budget. Per-launch DeviceBuffer alloc + H2D, dropped
  after launch.
- New bus_v3 launcher takes a `d_ops` pointer instead of relying on
  cuModuleGetGlobal + cuMemcpyHtoD to a __constant__ symbol.
- Kernel-side eval_affine_arg now walks an `op.value` AffineArg with
  #pragma unroll over MAX_TERMS_PER_ARG with `if (t < n_terms)` guard.
  Uniform per-warp so no divergence; ptxas DCEs unused term loads.
- EMITTER_VERSION bumped to 6.

Validation:
- 14 cuda unit tests pass including new partition_var_range_multi_term_affine
  exercising the 3-column timestamp-delta shape.
- Real APC=30 keccak input=10000 produces public_values_commit
  bit-identical to bytecode-VM baseline.

Performance impact (RTX 5090 warm cache):
- bus_kernel total: 315 ms (v5) → 302 ms (v6), 4% improvement
- fallback_vm: 256 ms (v5) → 213 ms (v6), 17% reduction
- Unhandled per APC: 23 → 1 (the non-affine outlier)

Smaller-than-projected gain because:
1. Per-row work increased ~30 ms (multi-term unroll)
2. The 1 remaining outlier is *very* expensive in bytecode VM
   (5.92 ms per APC where it appears, vs 0.12 ms/interaction baseline).
   It's quadratic (Reference*Reference) so multi-term affine can't catch
   it; only constraint-level derived-column materialization could.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds an alternative emit path that bakes per-op constants as immediates
in source instead of reading them from a runtime op table. Each APC
gets unique kernel sources per bus-kind; the PTX disk cache dedupes
across runs/APCs with identical partitions.

Implementation:
- emit_codegen_var_range / emit_codegen_tuple2 / emit_codegen_bitwise:
  emit per-op switch case bodies with mult, guard_col, max_bits/sizes,
  and affine coefs all baked as `Xu` literals.
- AffineArg / VarRangeOp / Tuple2Op / BitwiseOp now derive Hash so the
  source-hash key includes op data → identical partitions hit the cache.
- New v4 launchers (powdr_nvrtc_launch_bus_v4 + _bitwise) drop the
  d_ops pointer arg since constants are in source.
- Gated by env var POWDR_BUS_CODEGEN=1; default still uses the
  shared-source interpreter.

Per-stage timer added: bus_kernel_nvrtc_codegen.codegen for source
emission cost.

Validation:
- All 14 cuda unit tests still pass.
- Real APC=30 keccak input=10000 (cold + warm) produces
  public_values_commit bit-identical to the bytecode-VM baseline.

Performance vs interpreter (v6) — APC=30 keccak input=10000, RTX 5090:

  Stage              Interpreter      Codegen        Delta
  partition           1.16 ms         1.18 ms        +0.0
  codegen (host)      —               3.08 ms        +3.1
  kernel_prefetch     1.40 ms        40.8  ms       +39
  launch_var         27.3 ms         21.4  ms        -5.9
  launch_br          29.5 ms         17.0  ms       -12.5
  launch_bx          23.0 ms         14.5  ms        -8.5
  fallback_vm       213.3 ms        213.8  ms        +0.5
  TOTAL             302   ms        317   ms       +15

Sum of launch_* (just NVRTC kernel work):
  Interpreter: 79.8 ms     Codegen: 52.9 ms     -34%

Codegen kernels ARE faster per-launch (-34% on captured ops), validating
the constant-folding hypothesis. But per-APC source rebuild +
in-memory cache lookup for unique kernels overshoots the gain (+39ms
prefetch). Net bus_kernel total ends up ~5% slower than interpreter.

Cold first prove: 117s (compiles ~120 unique kernels, parallel via
rayon; longest single compile = 35s for a 357KB var_range source).
Subsequent warm proves: 8.4s (PTX disk cache hit for all kernels).

Open issues to address next:
1. Hoist kernel_prefetch out of per-APC loop. Most APCs are unique so
   the prefetch repeats source emission + hash + cache lookup 90 times
   per prove. A "compile all APCs' kernels once" pass at start of prove
   would eliminate this.
2. The 213 ms fallback_vm tail is unchanged — same quadratic outlier
   that codegen can't capture (it requires bilinear, not affine).
3. Source emission is 3 ms per APC; minor but optimizable (use Vec<u8>
   write_all instead of String + format!).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ic tail

Adds degree-≤2 polynomial decoding for arg expressions. Bilinear ops are
emitted in the same per-APC codegen kernels as affine ops (no kernel-side
runtime branch) — each switch arm is shape-specialized at emit time, so
affine cases stay at ~15-20 SASS while only bilinear cases pay the
cross-product cost.

Implementation:
- BilinearMonomials struct: (constant, linear[], bilinear[]) in canonical
  field form. Add/Sub/Mul/Negate operations that reject degree > 2.
- to_bilinear_monomials: recursive walker that distributes products and
  collapses to monomial form.
- decode_bilinear_arg: top-level entry that returns BilinearMonomials.
- VarRangeOpBilinear / Tuple2OpBilinear / BitwiseOpBilinear: new op
  types holding the full bilinear arg (no fixed-size struct, vecs).
- partition_apc_bus tries affine first, falls back to bilinear; keeps
  the affine fast path unchanged.
- emit_bilinear_inline: emits constant + linear + bilinear (with cell-
  load dedup across linear/bilinear references to the same column).
- emit_codegen_var_range/tuple2/bitwise now take both affine and
  bilinear op vecs and emit them in a single switch (affine cases first,
  then bilinear cases). Per-op shape specialization → no runtime branch.
- Interpreter path: bilinear ops added to the unhandled list (interpreter
  doesn't support bilinear in-kernel).
- EMITTER_VERSION bumped to 7.

Validation:
- All 14 cuda unit tests pass.
- Real APC=30 keccak input=10000 (cold + warm) produces
  public_values_commit bit-identical to the bytecode-VM baseline.

Performance vs prior backends (RTX 5090 warm cache, APC=30 keccak input=10000):

                              VM     Interpreter   Codegen v7   Codegen v8
  bus_kernel total          281 ms     302 ms       317 ms      141 ms ★
    fallback_vm                —       213 ms       214 ms       35 ms
    kernels (launch_*)         —        80 ms        53 ms       55 ms
    overhead (prefetch+...)    —         9 ms        50 ms       50 ms
  App proof end-to-end      8.24 s    8.35 s        8.42 s      8.30 s

bus_kernel: 50% reduction vs VM baseline; 55% reduction vs prior codegen.
fallback_vm: 84% reduction (213→35 ms). Per-APC fallback call count
dropped from 36 → 21 (bilinear caught 15 of the previously-unhandled
heavy interactions per prove).

Captured by NVRTC (work fraction):
  - Interpreter:    ~27% — fallback_vm dominated
  - Codegen v7:     ~30% — same residual
  - Codegen v8:     ~88% — bilinear closed the gap

Remaining 35 ms residual is the genuinely non-degree-2 expressions
(currently 21 fallback firings out of 90 trace_gen calls). Could be
pushed further with derived-column materialization at the
constraint-system level, but the bus_kernel is now smaller than the
VM baseline.

Cold first prove (PTX cache miss): 105s for APC=30. Subsequent runs
warm: 8.30s. The 40 ms kernel_prefetch overhead persists on every
prove because we rebuild + hash sources per APC; hoisting prefetch out
of the per-APC loop would drop bus_kernel another 40 ms → ~100 ms total.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds a `OnceLock<Arc<CachedBusCodegenKernels>>` field to
PowdrTraceGeneratorGpu that holds the compiled per-APC bus kernels.
On first use the cache is built (partition + codegen + NVRTC compile);
subsequent proves of the same APC pay only an atomic OnceLock load.

When `POWDR_JIT_BACKEND_BUS=nvrtc` and `POWDR_BUS_CODEGEN=1` are both
set, the cache is eagerly populated in `PowdrTraceGeneratorGpu::new()`
so the compile cost is paid at construction (process startup) instead
of inside any prove timer. The kernel sources are APC-deterministic
and `apc_height`-independent (height is a runtime kernel arg), so the
cache is height-agnostic and reusable across proves of any size.

Sub-stage timers added/refactored:
- bus_kernel_nvrtc_codegen.cache_get: per-call OnceLock load
- bus_kernel_nvrtc_codegen.cache_build.partition / .codegen / .compile:
  one-time-per-APC cost, run from inside cache_get on lazy init OR
  warm_cache on eager init
- bus_kernel_nvrtc_codegen.warm_cache: total eager-build wrapper

Performance — APC=30 keccak input=10000, RTX 5090, warm cache:

                                    VM        Interp   Codegen-v8   v9-eager
  App proof                      8.24 s        8.35 s       8.30 s    8.04 s
  bus_kernel total               281 ms        302 ms       141 ms     90 ms
    partition+codegen+prefetch       —             46           47       0.1
    launch_*                         —             80           55        55
    fallback_vm                      —            213           35        35

Speedups:
- vs VM baseline:      281 → 90 ms = 3.1× faster bus pass
- vs Interpreter:      302 → 90 ms = 3.3× faster
- vs Codegen v8:       141 → 90 ms = 1.6× faster (per-prove cache wins)

Captured by NVRTC: 99.5% by count, ~88% by work. Remaining 35 ms is
21 fallback_vm firings on degree-greater-than-2 expressions.

Validation: 14 cuda unit tests pass; APC=30 keccak input=10000 produces
public_values_commit bit-identical to bytecode-VM baseline.

Notes:
- warm_cache is per-APC; for batch proving of N proves on the same APC
  set, cost amortizes to ~0 per prove.
- Cold first-process compile: 105 s for APC=30 (compiles ~120 unique
  per-APC kernels). PTX disk cache makes subsequent process startups
  ~50 ms.
- Future: serializing PTX into the APC artifact format (autoprecompiles
  crate) would eliminate the cold-compile cost across processes too.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>